Synchronizing Broadcast Timeline Metadata

ABSTRACT

In one embodiment, a first content item having synchronization information, the first content item being a first instance of a reference content item, a candidate content item, which is a second instance of the reference content item, having synchronization information, a synchronizing processor which determines a synchronization point between the first content item and the candidate content item, on the basis of matching the synchronization information of the first content item and the candidate content item, a timeline metadata transmitter, which transmits stored timeline metadata to a device on which the first content item is playing, the transmission of the stored timeline metadata beginning from the synchronization point, wherein the stored timeline metadata includes timeline metadata that has been previously aggregated from earlier broadcasts of instances of the reference content item. Related systems, apparatus, and methods are also described.

FIELD OF THE INVENTION

The present invention generally relates to methods and systems forsynchronizing broadcast timeline metadata.

BACKGROUND OF THE INVENTION

Pay TV users typically rarely watch the production media-content (i.e.information and experiences of value to an end-user/audience) in itsoriginal form, rather various instances of the original content is whatmost consumers typically view. The term “instance”, in all of itsgrammatical forms, as used in the present specification and claims,refers to the version of the content with interleaved channel specificpromotional material and advertisement patterns—that keeps changingtime-to-time across regions and channels for the same content.

Multiple instances of the same content are broadcast and oftenre-broadcast on different channels or services (for example, and withoutlimiting the generality of the foregoing, a “+1 channel”, a catch-upservice, and so forth) that can also be recorded to TV terminals such asset-top box, personal video recorders or over-the-top (OTT) enabledcompanion devices by the consumers.

Intra-content (timeline) metadata (i.e. metadata associated with a pointin time inside the content) can be associated with any content, eitherautomatically by analyzing the content in real time, or by aggregatingreal time feeds, like social feeds or 3rd party feeds, e.g. sportsstatistics during a live game.

A TV viewer may choose to watch an instance of content live from thebroadcast or may decide to view it in a non-real time fashion, eitheroffline (i.e. playback from DVR) or through an alternative on-lineoption (e.g., catch-up service, +1 channel or directly from contentprovider's portal). But in any of these cases the viewer typically wouldlike to receive the same timeline metadata as if the viewer was watchingthe content during its original broadcast, when it is possible thattimeline metadata will be gathered in quality and quantity. TV consumersneed to have access to continuously growing timeline metadata of thecontent irrespective of whether the playback on their devices is fromlive broadcast or from personal video recorders.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated more fully fromthe following detailed description, taken in conjunction with thedrawings in which:

FIG. 1 is a simplified pictorial illustration of a system for contentdistribution, where different instances of a content item aredistributed to various client devices, constructed and operative inaccordance with an embodiment of the present invention;

FIG. 2 is a depiction of three different instances of the content itemfrom the system of FIG. 1, showing different ad patterns;

FIG. 3 is a block diagram depicting interaction of a client media deviceand a timeline metadata aggregator in the system of FIG. 1;

FIG. 4 is a block diagram depicting the client media device for use inthe system of FIG. 1;

FIG. 5 is a partial Venn diagram, partial flowchart depiction of alock-on mechanism as used in the system of FIG. 1; and

FIG. 6 is a simplified flowchart of a method of operation of the systemof FIG. 1.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

A system, apparatus, and method for synchronizing timeline metadataassociated with different instances of a content item is described. Thesystem, apparatus, and method comprising a first content item havingsynchronization information, the first content item being a firstinstance of a reference content item, a candidate content item, which isa second instance of the reference content item, having synchronizationinformation, a synchronizing processor which determines asynchronization point between the first content item and the candidatecontent item, on the basis of matching the synchronization informationof the first content item and the candidate content item, a timelinemetadata transmitter, which transmits stored timeline metadata to adevice on which the first content item is playing, the transmission ofthe stored timeline metadata beginning from the synchronization point,wherein the stored timeline metadata comprises timeline metadata thathas been previously aggregated from earlier broadcasts of instances ofthe reference content item. Related systems, apparatus, and methods arealso described.

Exemplary Embodiment

Reference is now made to FIG. 1, which is a simplified pictorialillustration of a system 100 for content distribution, where differentinstances of a content item 140 are distributed to various clientdevices 110, constructed and operative in accordance with an embodimentof the present invention. FIG. 1 depicts a typical pattern of flow ofcontent (not depicted) from a content provider 120 to the client devices110. The content provider 120 comprises a person or organization whichsupplies the content itself for use on the client devices 110.Typically, the content provider 120 provides the content to one or moreservice providers 130. The service provider 130 typically comprises acompany that provides its subscribers access to the content. The serviceprovider 130 might provide access to the content via a satellitebroadcast, via cable, or over an IP-enabled connection. Additionally oralternatively, the service provider 130 might be providing contentthrough a content distribution network, as is known in the art, orthrough multicast or unicast. All of the above mentioned methods ofcontent distribution are referred to in the present specification as“broadcast”.

For ease of discussion, the mechanisms and pathways used by the serviceprovider 130 will be referred to hereinafter as “channels”. The term issimilar to broadcast television channels, but is meant to be inclusiveof other mechanism, such as “over-the-top” content distributionpathways, broadcast distribution pathways, or other pathways known inthe art by which content is distributed by content providers to thevarious client devices 110.

Typically, several instances of content 140 might be prepared fordelivery to the various client devices 110. For example, and withoutlimiting the generality of the foregoing, one instance of the content140 may be prepared to be delivered via a channel which mainly servesconsumers in the Spanish language, while a second instance of theidentical content item may be prepared to be delivered via a channelwhich mainly serves consumers in the English language. Alternatively,one instance of a content item may be prepared with a particular set ofaccompanying promotional material (i.e. promos) and advertisements,while a second instance of a content item may be prepared with adifferent set of promos and advertisements.

Reference is now made to FIG. 2, which is a depiction of three differentinstances of the content item from the system of FIG. 1, showingdifferent ad patterns. Three different instances 240A, 240B and 240C ofthe content item are depicted, each represented as a rectangular block.Within each of the rectangular blocks there are differing ad patterns.An additional consideration is that because of its particular adpattern, instance 240C is actually, in total, longer than correspondinginstances 240A and 240B.

The ad patterns, as noted above, typically comprise a number ofdifferent elements: advertisements for specific products; channelspecific promos; information about current reality shows; and so forth.

By way of example, a first instance 240A of a content item may bebroadcast close to the time when the content item is relatively new. Forexample, a premier of a first episode of a new television mini-serieswill be broadcast once as a premier, may be re-broadcast an hour lateron a +1 channel, and may still be available on various over-the-topservices or channels within a week of the premier broadcast. As such,there will typically be a lot of social media related timeline metadatagenerated concerning the premier and the various repeats of the premieron +1 channels, OTT services, and so forth. Additionally, advertisementsand promos in the ad patterns associated with the premier will becurrent, relating to the goods and services on sale at that time, or tocurrent events and other current television programs.

By contrast, an instance of the same content item broadcast a year afterthe premier episode will typically generate little timeline metadata onsocial media networks. Advertisements and promos in the ad patternsassociated with the broadcast a year later will also be current.However, because the content in the ad pattern of the rebroadcast willbe current, the ad pattern will not match the content in the ad patternfrom the premier.

It is appreciated, in view of the above discussions of FIGS. 1 and 2that the content item 240A, 240B, 240C may be recorded on a storageelement on the client device 110 (the storage element may compriseeither fixed or removable storage), may be delivered over a broadcastnetwork, may be delivered over the cloud, or over any other appropriatecontent delivery mechanism and/or network, as is known in the art.

Reference is now made to FIG. 3, which is a block diagram depictinginteraction of a client media device and a timeline metadata aggregatorin the system of FIG. 1. A media device 300 is depicted with a contentitem having a timeline 310, the content item being played at present onthe media device 300, and hence is labeled as “Playing Now”. Variouspoints of broadcast timeline metadata in a candidate content item 320are being compared 325. If a sufficient number of the broadcast timelinemetadata points match, then the candidate content item 320 and the nowplaying content item 310 are deemed to be different instances of thesame content item. Broadcast instances of a content item are typicallydifferent than corresponding DVD versions of the same content item interms of interleaved promos, advertisements, and so forth, and it islikely that there may be scene-cuts (across various regions) whichresults in missing subtitles or video fingerprints. In such situations,percentage matches somewhere between 80-100% may be sufficient toconclude that the candidate item is a different instance of the samecontent. It is appreciated that the candidate content item 320 may bethought of as a reference content item for any other instances of thecontent item to which it is being compared as it is stored where thecomparisons between the candidate content item 320 and the now playingcontent item 310 is affected.

The comparison is performed by a metadata aggregator and synchronizationunit 330. More specifically, the metadata aggregator and synchronizationunit 330 comprises at least one processor 340, and may comprise morethan one processor 340. One of the processors 340 may be a specialpurpose processor operative to perform comparison and synchronization ofthe broadcast timeline metadata, according to the method describedherein. In addition, the metadata aggregator and synchronization unit330 comprises non-transitory computer-readable storage media (i.e.memory) 350. The memory 350 may store instructions, which at least oneof the processors 340 may execute, in order to perform the method oftimeline metadata comparison and synchronization described herein.Metadata aggregator and synchronization unit 330 also comprises storage360. The storage 360 stores broadcast timeline metadata which has beenpreviously aggregated from earlier broadcasts of various content items,the aggregated metadata being used for comparisons with candidatecontent items 320.

Metadata aggregator and synchronization unit 330 also comprises othertypical and standard hardware and software components as are known inthe art. Additionally, the metadata aggregator and synchronization unit330 also comprises a transmitter 370 which, once processor 340determines that the candidate content item 320 and the now playingcontent item 310 are different instances of the same content item,transmits stored broadcast content metadata from storage 360 to areceiver 380 comprised in the media device 300. The media device 300 isthen able to make the received broadcast content metadata available to auser of the media device 300.

The metadata aggregator and synchronization unit 330 treats differentinstances of the same content as different entities and attempts tomatch against previously stored timeline metadata in real-time. This waythe system synchronizes current broadcast instance with already archivedcontent. The metadata aggregator and synchronization unit 330 isconstantly comparing the metadata it is creating against already knowntimeline metadata to find potential matches. It continues comparingtimeline metadata against the already matched stored content to gain aconfidence measure that the matches are the same content—this is calledthe “lock-on mechanism”.

It is appreciated that there are a variety of major sub-systems whichhave dedicated functionality in order to achieve the lock-on-mechanism.The first sub-system analyzes audio/video streams and generatesprimary/direct metadata (like subtitles, faces, audio-meter etc.). Next,based on these primary metadata types (either the same or a different)sub-system extracts secondary metadata (likescene/personality/object/place details, and so forth, as is known in theart). Ideally, this sub-system should not generate metadata if themetadata has been generated in the past—only the new candidate metadatagets appended to previously stored version. Furthermore, all of thesetimeline/intra-content metadata continuously get streamed in real-timeinto another sub-system which aggregates and archives them into storage(with pre-defined data-model). Various client devices get connected andrequest timeline metadata from this sub-system.

Once the lock-on mechanism determines that a sufficient confidencemeasure has been achieved and a lock has been acquired, the system hassynchronized the original videos with the currently playing one and canprovide additional timeline metadata from the matched content. The pointin the content item from where the lock is acquired may be viewed as a“synchronization point”. The currently playing content item may beconsidered an instance of a reference content item to which othercontent items may be compared.

It is appreciated that the technique of achieving a lock variesdepending on the type of timeline metadata which has been used toachieve the lock. By way of example, in the case of subtitles, straightforward text match is used—out of <n> subsequent matches if the matchescontinue to be somewhere between 80-100% (say), the lock remains. If thepercentage falls below 80%, the lock has to be acquired again—and theprocess continues. From then on, the system will keep checking that thelock is still correct—when the lock is lost, the process is restarted.

It is appreciated that the process of acquiring a lock is a continuousprocess that will build confidence overtime. As such, it is appropriatethat the system not use a 100% threshold in order to cater to minordifferences in the metadata that are artifacts of the extractionprocess, such as subtitles which may be a little bit different, a fewletters or words not matching, and so forth. Also, issues known in theart which may be introduced when performing OCR (optical characterrecognition) of the received subtitles should also be allowed for. Assuch, an implementation decision may allow for the experience to giveincorrect metadata for few seconds after lock has been lost (in reality)rather than cutting the metadata feed too early.

Reference is now made to FIG. 4, which is a block diagram depicting theclient media device 400 for use in the system of FIG. 1. Typicalimplementations of the client media device 400 include, but are notlimited to a tablet device, smartphone, desktop or portable computer,set-top box, Internet-enabled television, media center PC, or any othersuitable device, such as is known in the art.

The client media device 400 comprises at least one processor 410, a userinterface (typically a graphical user interface, GUI) 420, and a mediaplayer 430. The GUI 420 and the player 430 may comprise a singleapplication, may be two applications which interact with each other, orthe GUI may be part of a different application, such as a Web browser.The GUI 420 enables a user of the client media device 400 to interactwith the player 430, request content, start, stop, and pause the player430, and perform other well-known typical user interactions with theplayer 430. The player 430 typically outputs played media (i.e. contentitems) on a display 440, such as a screen.

The client media device 400 may comprise more than one processor 410. Inaddition, the client media device 400 comprises non-transitorycomputer-readable storage media (i.e. memory) 450. The memory 450 maystore instructions, which at least one of the processors 410 mayexecute. Client media device 400 also comprises storage 460, on which,for example, the currently playing content item 310 (FIG. 3) is stored.The client media device 400 also comprises typical and standard hardwareand software components as are known in the art.

The client media device 400 further comprises communications ports anddrivers 470, included among them the receiver 380 (FIG. 3).

Reference is now made to FIG. 5, which is a partial Venn diagram,partial flow chart depiction of a lock-on mechanism as used in thesystem of FIG. 1. By way of a non-limiting example illustrating theacquisition of the lock-on process described above, assume a scenario ofa subscriber playback of an instance of a program recorded from achannel different than the one used by the metadata aggregator andsynchronization unit 520 to perform content timeline metadata indexing

A good candidate of timeline metadata to illustrate this scenario is theuse of widely available subtitles in broadcast content. For example, aset top box (STB—media device) 500 is able to extract the subtitles 510from its recorded content and provide them to the metadata aggregatorand synchronization unit 520, which can then make use of them to acquirea lock on the previously stored content and can then provide the STB 500with the other associated timeline metadata already stored—allsynchronized with the video.

Expanding on this example, a set of timeline metadata, depicted as “nowplaying metadata” 540 is available to the STB 500, as timeline metadatawhich accompanies the now playing content item 310 (FIG. 3). Similarly,the candidate content item 320 (FIG. 3) of the metadata aggregator andsynchronization unit 520 comprises its own accompanying timelinemetadata 550, which may comprise metadata aggregated from socialnetworks (or other Internet based sources) 552, secondary metadata(defined below) 554, and subtitles 556, as well as other types oftimeline metadata (not depicted). The subtitles 510 from the STB 500 andthe subtitles 556 from the candidate content item metadata are inputinto the processor 340 (FIG. 3), which determines if the two differentsets of subtitles 510, 556 match or do not match (step 560).

It is appreciated that it is highly likely that there are other metadataitems among the now playing metadata 540 besides the subtitles 510.

If the two different sets of subtitles 510, 556 do not match (step 563),then the system does not acquire a lock-on between the timelines of thecandidate content item 320 (FIG. 3) and the now playing content item 310(FIG. 3). Another candidate content item 320 is sought in the metadataaggregator and synchronization unit 520 database 360 (FIG. 3), and theprocess of determining if the two content items match or not (i.e. areor are not different instances of the same content item) repeats (step560).

However, if Step 560 determines that the two different sets of subtitles510, 556 do match (step 568) (i.e. are different instances of the samecontent item), then a lock-on between the two timelines is acquired(step 580). The metadata aggregator and synchronization unit 520 thenprovides timeline metadata which is associated with the candidate (now aconfirmed match) content item 320 (FIG. 3) to the STB 500 (step 590).

There exists a possibility that both quality and quantity of thedelivered timeline metadata vary across multiple playback of the samerecording. This is because the metadata aggregator and synchronizationunit 520 continuously keeps enriching the stored replica of the contentwith new set of available timeline metadata from various sources. It isappreciated that this is the reason that the candidate metadata Venndiagram 550 is depicted substantially larger than the now playingmetadata Venn diagram 540.

An alternative method which may be used to establish the lock-on wouldbe the use of video fingerprint technology at client device. The clientdevice samples video at periodic interval (say every 3-5 seconds) andextracts a frame to compute video-fingerprint. Thesevideo-fingerprint/signatures are provided to the remote metadataaggregator and synchronization unit 520. The metadata aggregator andsynchronization unit 520 make use of these received video-fingerprintsto acquire a lock on the previously stored content and can then providethe client-device (like STB) with the other associated timeline metadataalready stored—all synchronized with the video. Generally videofingerprinting technologies are resolution agnostic (to certainextent)—the client-devices (like STBs) which are limited in computationcapabilities can sample the video with lower resolution to reducecomputation overhead.

It is appreciated that there are other methods of acquiring lock-on. Themethods described above are two examples of such methods.

It is also appreciated that the selection of a candidate content item320 may depend on various factors. For example, if, upon examination ofbroadcast metadata it is determined that an instance of a content itemhas the same: title field, starring actors and directors (by way ofexample) as the now playing 310 (FIG. 3) content item, then the contentitem is deemed to be an appropriate candidate content item 320 (FIG. 3).On the other hand, if the various fields which may be used, for example,to populate a program guide about a content item do not match, then theinstance of a content item having the different: title field, starringactors and directors (by way of example) as the now playing 310 (FIG. 3)content item, is most likely a different instance of the same contentitem. Therefore, the content item with non-matching program guide typefields would not be deemed a likely candidate content item 320 (FIG. 3).

The discussion of FIG. 5 above mentioned secondary metadata 554. One wayof classifying the timeline metadata would be to differ between metadatawhich is immediately related to the content item. By way of example, ifthe subtitles mention New York City, then aside from the primarymetadata of the subtitles, a Wikipedia article about New York City,various tourist websites about New York City, and so forth, may becollected by an off-line secondary metadata aggregator. When thereference to New York City occurs in the now playing content item 310(FIG. 1), the timeline metadata will also have the collected secondarymetadata which may then be presented to a user of the STB 500.

It is appreciated that the present invention does not rely on pre-addedfingerprints to the audio/video for synchronization purposes but workson any content as it is distributed at the time of filing of thisapplication. For example, fingerprinting 594, 595, as described above,performed on the content being rendered on media device 500, may becompared to aggregated fingerprints at the metadata aggregator andsynchronization unit 520. The comparing of the fingerprints from themedia device 500 and the metadata aggregator and synchronization unit520 would then follow the procedure beginning with step 560, todetermine if the two sets of fingerprints 594, 595 match or do notmatch.

The media device 500 may be configured to execute either one or both ofthe above processes, i.e. matching subtitles 510, 556 and/orfingerprints 594, 595. The lock-on mechanism may operate in amulti-modal manner. This provides a failsafe mechanism, so that iflock-on subtitles 510, 556 fails, fingerprint 594, 595 based lock maycontinue and vice-versa.

The lock-on failsafe mechanism described above is helpful in situationswhere subtitles 510, 556 are being extracted using OCR techniques eitherfrom DVB-subtitle streams or from video frame directly (because, forexample, some regions do not have subtitles coming along withaudio/video components for most of the channels).

Reference is now made to FIG. 6, which is a simplified flowchart of amethod of operation of the system of FIG. 1. The method of FIG. 8 isbelieved to be self-explanatory with reference to the above discussion.

It is appreciated that software components of the present invention may,if desired, be implemented in ROM (read only memory) form. The softwarecomponents may, generally, be implemented in hardware, if desired, usingconventional techniques. It is further appreciated that the softwarecomponents may be instantiated, for example: as a computer programproduct or on a tangible medium. In some cases, it may be possible toinstantiate the software components as a signal interpretable by anappropriate computer, although such an instantiation may be excluded incertain embodiments of the present invention.

It is appreciated that various features of the invention which are, forclarity, described in the contexts of separate embodiments may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention which are, for brevity, described in thecontext of a single embodiment may also be provided separately or in anysuitable subcombination.

It will be appreciated by persons skilled in the art that the presentinvention is not limited by what has been particularly shown anddescribed hereinabove. Rather the scope of the invention is defined bythe appended claims and equivalents thereof:

What is claimed is:
 1. A system for synchronizing timeline metadata associated with different instances of a content item, the system comprising: a first content item having synchronization information, the first content item being a first instance of a reference content item; a candidate content item, which is a second instance of the reference content item, having synchronization information; a synchronizing processor which determines a synchronization point between the first content item and the candidate content item, on the basis of matching the synchronization information of the first content item and the candidate content item; a timeline metadata transmitter, which transmits stored timeline metadata to a device on which the first content item is playing, the transmission of the stored timeline metadata beginning from the synchronization point, wherein the stored timeline metadata comprises timeline metadata that has been previously aggregated from earlier broadcasts of instances of the reference content item.
 2. The system according to claim 1 wherein the synchronization information comprises a subset of the timeline metadata.
 3. The system according to claim 1 wherein the synchronization information comprises subtitles.
 4. The system according to claim 1 wherein the synchronization information comprises a video fingerprint.
 5. The system according to claim 1 wherein the timeline metadata comprises metadata aggregated from social networks.
 6. The system according to claim 1 wherein the timeline metadata comprises secondary metadata.
 7. The system according to claim 1 wherein the timeline metadata comprises subtitles.
 8. The system according to claim 1 wherein the timeline metadata transmitter further comprises a metadata aggregator.
 9. The system according to claim 8 wherein the metadata aggregator aggregates metadata available on the Internet.
 10. The system according to claim 8 wherein the metadata aggregator aggregates metadata from a plurality of instances of the reference content item.
 11. A method for synchronizing timeline metadata associated with different instances of a content item, the method comprising: providing a first content item, the first content item having synchronization information, the first content item being a first instance of a reference content item; providing a candidate content item, which is a second instance of the reference content item, the candidate content item having synchronization information; determining, at a synchronizing processor, a synchronization point between the first content item and the candidate content item, on the basis of matching the synchronization information of the first content item and the candidate content item; transmitting timeline metadata by a timeline metadata transmitter, which transmits stored timeline metadata to a device on which the first content item is playing, the transmission of the timeline metadata beginning from the synchronization point, wherein the stored timeline metadata comprises timeline metadata that has been previously aggregated from earlier broadcasts of instances of the reference content item.
 12. The method according to claim 11 wherein the synchronization information comprises a subset of the timeline metadata.
 13. The method according to claim 11 wherein the synchronization information comprises subtitles.
 14. The method according to claim 11 wherein the synchronization information comprises a video fingerprint.
 15. The method according to claim 11 wherein the timeline metadata comprises metadata aggregated from social networks.
 16. The method according to claim 11 wherein the timeline metadata comprises secondary metadata.
 17. The method according to claim 11 wherein the timeline metadata comprises subtitles.
 18. The method according to claim 11 wherein the timeline metadata transmitter further comprises a metadata aggregator.
 19. The method according to claim 18 wherein the metadata aggregator aggregates metadata available on the Internet.
 20. The method according to claim 18 wherein the metadata aggregator aggregates metadata from a plurality of instances of the reference content item. 